Multiple Hover Point Gestures

ABSTRACT

Example apparatus and methods concern detecting and responding to a multiple hover point gesture performed for a hover-sensitive device. An example apparatus may include a hover-sensitive input/output interface configured to detect multiple objects in a hover-space associated with the hover-sensitive input/output interface. The apparatus may include logics configured to identify an object in the hover-space, to characterize an object in the hover-space, to track an object in the hover-space, to identify a multiple hover point gesture based on the identification, characterization, and tracking, and to control a device, application, interface, or object based on the multiple hover point gesture. In different embodiments, multiple hover point gestures may be performed in one, two, three, or four dimensions. In one embodiment, the apparatus may be event driven with respect to handling gestures.

BACKGROUND

Users of smart phones, tablets, and other touch devices are familiarwith touching the screen of the device to cause the device to perform anaction. The touch action generally simulates a mouse click or buttonpress. Conventionally, touch-sensitive screens have also supportedgestures where one or two fingers were placed on the touch-sensitivescreen then moved in an identifiable pattern. For example, users mayinteract with an input/output interface on the touch-sensitive screenusing gestures like a swipe, a pinch, a spread, a tap or double tap, orother gestures. Conventionally, the touch-sensitive screen had a singletouch point, or a pair of touch points for gestures like a pinch.

Devices like smart phones and tablets may also be configured withscreens that are hover-sensitive. Hover-sensitive screens may rely onproximity detectors to detect objects that are within a certain distanceof the screen. Conventional hover-sensitive screens detected singleobjects in a hover-space associated with the hover-sensitive device andresponded to events like a hover-space entry event or a hover-space exitevent. Conventional hover-sensitive devices typically attempted toimplement actions that were familiar to users of touch-sensitivedevices. When presented with two or more objects in a hover-space, ahover-sensitive device may have identified the first entry as being thehover point and may have ignored other items in the hover-space.

Some devices may have screens that are both touch-sensitive andhover-sensitive. Conventionally, devices with screens that are bothtouch-sensitive and hover-sensitive may have responded to touch eventsor to hover events. While a rich set of interactions may be possibleusing a screen in a touch mode or a hover mode, this binary approach mayhave limited the richness of the experience possible for an interfacethat is both touch-sensitive and hover-sensitive. Some conventionaldevices may have responded to gestures that started with a touch eventand then proceeded to a hover event. Limiting interactions to require aninitiating touch may have needlessly limited the user experience. Somedevices with screens that are both touch-sensitive and hover-sensitivemay have interacted with a single touch point or a single hover point.Limiting interactions to a single touch or hover point may have limitedthe richness of the experience possible to users of devices. Someconventional devices may have responded to hover gestures that were tiedto an object displayed on the screen. For example, hovering over adisplayed control may have accessed the control. The control may thenhave been manipulated using a gesture (e.g., swipe up, swipe down).Limiting hover interactions to only operate on objects or controls thatare displayed on a screen may needlessly limit the user experience.

SUMMARY

This Summary is provided to introduce, in a simplified form, a selectionof concepts that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Example methods and apparatus are directed towards interacting with ahover-sensitive device using gestures that include multiple hoverpoints. A multiple hover point gesture may rely on a sequence orcombination of gestures to produce a different user interaction with ascreen that has hover-sensitivity. The multiple hover point gestures mayinclude a hover gather, a hover spread, a crank or knob gesture, a poofor explode gesture, a slingshot gesture, or other gesture. Byidentifying, characterizing, and tracking multiple hover points usingthe hover capability provided by an interface that is hover-sensitive,example methods and apparatus provide new gestures that may be intuitivefor users and that may increase productivity or facilitate newinteractions with applications (e.g., games, email, video editing)running on a device with the interface.

Some embodiments may include logics that detect, characterize, and trackmultiple hover points. Some embodiments may include logics that identifyelements of the multiple hover point gestures from the detection,characterization, and tracking data. Some embodiments may maintain astate machine and user interface in response to detecting the elementsof the multiple hover point gestures. Detecting elements of the multiplehover point gestures may involve receiving events from the userinterface. For example, events like a hover enter event, a hover exitevent, a hover approach event, a hover retreat event, a hover point moveevent, or other events may be detected as a user positions and movestheir fingers or other objects in a hover-space associated with adevice. Some embodiments may also produce gesture events that can behandled or otherwise processed by other devices or processes.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various example apparatus, methods,and other embodiments described herein. It will be appreciated that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one example of the boundaries. In someexamples, one element may be designed as multiple elements or multipleelements may be designed as one element. In some examples, an elementshown as an internal component of another element may be implemented asan external component and vice versa. Furthermore, elements may not bedrawn to scale.

FIG. 1 illustrates an example hover-sensitive device.

FIG. 2 illustrates an example state diagram associated with an examplemultiple hover point gesture.

FIG. 3 illustrates an example multiple hover point gather gesture.

FIG. 4 illustrates an example multiple hover point spread gesture.

FIG. 5 illustrates an example interaction with an examplehover-sensitive device.

FIG. 6 illustrates actions, objects, and data associated with a multiplehover point gesture.

FIG. 7 illustrates actions, objects, and data associated with a multiplehover point gesture.

FIG. 8 illustrates actions, objects, and data associated with a multiplehover point gesture.

FIG. 9 illustrates actions, objects, and data associated with a multiplehover point gesture.

FIG. 10 illustrates actions, objects, and data associated with amultiple hover point gesture.

FIG. 11 illustrates actions, objects, and data associated with amultiple hover point gesture.

FIG. 12 illustrates actions, objects, and data associated with amultiple hover point gesture.

FIG. 13 illustrates actions, objects, and data associated with amultiple hover point gesture.

FIG. 14 illustrates actions, objects, and data associated with amultiple hover point gesture.

FIG. 15 illustrates an example method associated with a multiple hoverpoint gesture.

FIG. 16 illustrates an example method associated with a multiple hoverpoint gesture.

FIG. 17 illustrates an example apparatus configured to support amultiple hover point gesture.

FIG. 18 illustrates an example apparatus configured to support amultiple hover point gesture.

FIG. 19 illustrates an example cloud operating environment in which anapparatus configured to interact with a multiple hover point gesture mayoperate.

FIG. 20 is a system diagram depicting an exemplary mobile communicationdevice configured to interact with a user through a multiple hover pointgesture.

FIG. 21 illustrates an example z distance and z direction in an exampleapparatus configured to process a multiple hover point gesture.

FIG. 22 illustrates an example displacement in an x-y plane and in a zdirection from an initial point.

DETAILED DESCRIPTION

Example apparatus and methods concern multiple hover point gestureinteractions with a device. The device may have an interface that ishover-sensitive. FIG. 1 illustrates an example hover-sensitive device100. Device 100 includes an input/output (i/o) interface 110. I/Ointerface 110 is hover-sensitive. I/O interface 110 may display a set ofitems including, for example, a user interface element 120. Userinterface elements may be used to display information and to receiveuser interactions. Device 100 or i/o interface 110 may store state 130about the user interface element 120 or other items that are displayed.The state 130 of the user interface element 120 may depend on hovergestures. The state 130 may include, for example, the location of anobject displayed on the i/o interface 110, whether the object has beenbracketed, or other information. The state information may be saved in acomputer memory.

The device 100 may include a proximity detector that detects when anobject (e.g., digit, pencil, stylus with capacitive tip) is close to butnot touching the i/o interface 110. Hover user interactions may beperformed in the hover-space 150 without touching the device 100. Theproximity detector may identify the location (x, y, z) of an object(e.g., finger) 160 in the three-dimensional hover-space 150, where x andy are parallel to the proximity detector and z is perpendicular to theproximity detector. The proximity detector may also identify otherattributes of the object 160 including, for example, how close theobject 160 is to the i/o interface (e.g., z distance), the speed withwhich the object 160 is moving in the hover-space 150, the orientation(e.g., pitch, roll, yaw) of the object 160 with respect to the device100 or hover-space 150, the direction in which the object 160 is movingwith respect to the hover-space 150 or device 100 (e.g., approaching,retreating), a gesture (e.g., gather, spread) made by the object 160, orother attributes of the object 160. While conventional interfaces mayhave handled a single object, the proximity detector may detect morethan one object in the hover-space 150. For example, object 160 andobject 170 may be simultaneously detected, characterized, tracked, andconsidered together as performing a multiple hover point gesture.

In different examples, the proximity detector may use active or passivesystems. For example, the proximity detector may use sensingtechnologies including, but not limited to, capacitive, electric field,inductive, Hall effect, Reed effect, Eddy current, magneto resistive,optical shadow, optical visual light, optical infrared (IR), opticalcolor recognition, ultrasonic, acoustic emission, radar, heat, sonar,conductive, and resistive technologies. Active systems may include,among other systems, infrared or ultrasonic systems. Passive systems mayinclude, among other systems, capacitive or optical shadow systems. Inone embodiment, when the proximity detector uses capacitive technology,the detector may include a set of capacitive sensing nodes to detect acapacitance change in the hover-space 150. The capacitance change may becaused, for example, by a digit(s) (e.g., finger, thumb) or otherobject(s) (e.g., pen, capacitive stylus) that comes within the detectionrange of the capacitive sensing nodes. In another embodiment, when theproximity detector uses infrared light, the proximity detector maytransmit infrared light and detect reflections of that light from anobject within the detection range (e.g., in the hover-space 150) of theinfrared sensors. Similarly, when the proximity detector uses ultrasonicsound, the proximity detector may transmit a sound into the hover-space150 and then measure the echoes of the sounds. In another embodiment,when the proximity detector uses a photo-detector, the proximitydetector may track changes in light intensity. Increases in intensitymay reveal the removal of an object from the hover-space 150 whiledecreases in intensity may reveal the entry of an object into thehover-space 150.

In general, a proximity detector includes a set of proximity sensorsthat generate a set of sensing fields in the hover-space 150 associatedwith the i/o interface 110. The proximity detector generates a signalwhen an object is detected in the hover-space 150. In one embodiment, asingle sensing field may be employed. In other embodiments, two or moresensing fields may be employed. In one embodiment, a single technologymay be used to detect or characterize the object 160 in the hover-space150. In another embodiment, a combination of two or more technologiesmay be used to detect or characterize the object 160 in the hover-space150.

In one embodiment, characterizing the object includes receiving a signalfrom a detection system (e.g., proximity detector) provided by thedevice. The detection system may be an active detection system (e.g.,infrared, ultrasonic), a passive detection system (e.g., capacitive), ora combination of systems. The detection system may be incorporated intothe device or provided by the device.

Characterizing the object may also include other actions. For example,characterizing the object may include determining that an object (e.g.,digit, stylus) has entered the hover-space or has left the hover-space.Characterizing the object may also include identifying the presence ofan object at a pre-determined location in the hover-space. Thepre-determined location may be relative to the i/o interface.

FIG. 2 illustrates a state diagram associated with supporting multiplehover point gestures. When a hover-sensitive apparatus detects multipleobjects (e.g., fingers, thumbs, stylus) in the hover-space associatedwith the apparatus, the detect state 210 associated with a multiplehover point gesture may be entered. Once multiple objects have beenidentified, the individual objects may be characterized on attributesincluding, but not limited to, position (e.g., x,y,z co-ordinates), size(e.g., width, length), shape (e.g., round, elliptical, square,rectangular), and motion (e.g., approaching, retreating, moving in x-yplane). The characterization may be performed when a hover point enterevent occurs and may be repeated when a hover point move event occurs.When two or more objects in the hover-space have been characterized,then the characterize state 220 may be achieved.

When at least one of the multiple hover points that were characterizedmoves, example apparatus and methods may track the movement of the hoverpoint. The tracking may involve relating characterizations that areperformed at different times. When at least one of the multiple hoverpoints that were characterized has been tracked, then the track state230 may be achieved. Once multiple hover points have been detected,characterized, and tracked, it may be possible to select a multiplehover point gesture based, at least in part, on the size, shape,movement, and relative movement of the hover points. For example,multiple hover points that move inwards towards each other may describea gather gesture while multiple hover points that move outwards fromeach other may describe a spread gesture. Multiple hover points thatrotate about a central point may describe a crank or knob gesture. Whenthe identification, characterization, and tracking data match a gesturepattern, then the select state 240 may be achieved.

Once the select state 240 has been achieved, actions that preceded theselection or actions that follow the selection may be evaluated todetermine what control to exercise during the control state 250. Duringthe control state 250, the multiple hover point gesture may cause theapparatus to be controlled (e.g., turn on, turn off, increase volume,decrease volume, increase intensity, decrease intensity), may cause anapplication being run on the device to be controlled (e.g., startapplication, stop application, pause application), may cause an objectdisplayed on the device to be controlled (e.g., moved, rotated, sizeincreased, size decreased), or may cause other actions.

FIGS. 3 and 4 illustrate multiple hover point gather and spread gesturesthat may be recognized by users of touch sensitive devices. Unlike theirtouch sensitive cousins, the gather and spread gestures may operate inone, two, three, or even four dimensions. A conventional pinch gesturebrings two points together along a single line. A multiple hover pointgather gesture may bring points together in an x/y plane, but may alsoreposition the points in the z direction at the same time. Aconventional pinch gesture requires a user to put two fingers onto aflat touch screen. This may be difficult if even possible to achievewhen the device is being held in one hand, when the device is just outof reach, when the device is oriented at an awkward angle, or for otherreasons. For example, a user's fingers and thumbs may be differentlengths. To perform a conventional touch screen pinch gesture, a usermay need to re-orient the device to accommodate the different lengths oftheir digits, or may need to tilt, rotate, lift, or otherwise manipulatetheir digits to match the flat touch screen. This may be extremelydifficult for a person with arthritis in their hands or fingers. Amultiple hover point gather gesture is not limited like the conventionalpinch gesture. A multiple hover point gather gesture is performedwithout touching the screen. The digits do not need to be exactly thesame distance from the screen. The gather may be performed without firstreferencing a particular object on a display by touching or otherwiseidentifying the object. Conventional pinch gestures typically requirefirst selecting an item or control and then pinching the object. Exampleapparatus and methods are not so limited, and may generate a gathercontrol event regardless of what, if anything, is displayed on a screen.

FIG. 3 illustrates an example multiple hover point gather gesture.Fingers 310 and 320 are positioned in an x-y plane 330 in thehover-space above device 300. While an x-y plane is described, moregenerally, fingers may be placed in a volume above device 300 and movedin x and y directions. Fingers 310 and 320 have moved together in thex-y plane or volume over apparatus 300. Finger 310 is closer to thehover-sensitive screen than finger 320. In one embodiment, exampleapparatus and methods may measure the distance from the screen to thefingers in the z direction. Apparatus 300 has identified hover points312 and 322 associated with fingers 310 and 320 respectively. As thefingers 310 and 320 move together, the hover points 312 and 322 alsomove together. When the hover points 312 and 322 have moved close enoughtogether, then a multiple hover point gather may be performed. Thegather gesture may be used to reduce screen brightness, to limit asocial circle with which a user interacts, to make an object smaller, tozoom in on a picture, to gather an object to be lifted, to crush avirtual grape, to control device volume, or for other reasons.

Unlike a conventional touch screen pinch gesture where only two pointsare brought together, example gather gestures may be extended to includea three, four, five, or more point gather gesture. Thus, rather thansimply bringing two points together along a single connecting line,example multiple hover point gather gestures may gather together itemsin a virtual area or volume, rather than collapsing points along a line.Thus, rather than simply pinching a single item represented in a flatspace on a display, a multiple hover point gather may grab multipleobjects represented in a three dimensional display. Additionally, ratherthan manipulating an object in just one dimension (e.g., linearlydecrease size of object pinched), example apparatus and methods maymanipulate an object in three dimensions. For example, a sphere or otherthree dimensional volume (e.g., apple) that is manipulated by a multiplehover point gather gesture may shrink spherically, rather than justlinearly. In one embodiment, the multiple hover point gather gesture maysimply bring two points together in an x/y plane along a singleconnecting line. Example apparatus and methods may perform the gathergesture without requiring interaction with a touch screen, withoutrequiring interaction with a camera-based system, and without referenceto any particular object displayed on device 300. Note that device 300is not displaying any objects. The gather gesture may be used withrespect to objects, but may also be used to control things other thanindividual objects displayed on device 300. Thus, example apparatus andmethods may operate more independently than conventional systems thatrequire touches, cameras, or interactions with specific objects.

FIG. 4 illustrates an example multiple hover point spread gesture.Fingers 310 and 320 have moved apart from each other. Thus,corresponding hover points 312 and 322 have also moved apart. Thisspread may be used to virtually release an object(s) that was pinched,lifted, and carried to a new virtual location. The location at which theobject will be placed on the display on apparatus 300 may depend, atleast in part, on the location of hover points 312 and 322. Unlike aconventional one dimensional spread gesture performed on a touch screen,the multiple hover point spread gesture may operate in three dimensions.Returning to the spherical object or apple example, multiple hoverpoints may be located inside the virtual sphere and then spread apart.The sphere may then expand in three dimensions instead of just linearlyin one direction. In one embodiment, since the apparatus may track the zdistance for the multiple hover points, and since the apparatus maytrack the rate at which the multiple hover points are moving apart, thespread gesture may be used, for example, to throw virtual dust in theair or fling virtual water off the end of fingertips. The volume coveredby the virtual dust throw may depend, for example, on the distance fromthe screen at which the spread was performed and the rate at which thespread was performed. For example, a spread performed slowly maydistribute the dust to a smaller volume than a spread performed morerapidly. Additionally, a spread performed farther from the screen mayspread the dust more widely than a spread performed close to the screen.

A conventional one dimensional spread may only enlarge a selected objectin a single dimension, while an example multiple hover point spreadoperating in three dimensions may enlarge objects in multipledimensions. The spread gesture may also be used in other applicationslike gaming control (e.g., spreading magic dust), arts and crafts (e.g.,throwing paint in modem art), industrial control (e.g., spraying avirtual mist onto a control surface), engineering (e.g., computer aideddrafting), and other applications. Unlike conventional touch spreadgestures that operate to change a single dimension of a single selecteditem, example apparatus and methods may operate on a set of objects inan area or volume without first identifying or referencing thoseobjects. Instead, a multiple hover point spread gesture may be used togenerate a spread control event for which an object, user interface,application, portion of a device, or device may subsequently be selectedfor control. While users may be familiar with the touch spread gestureto enlarge objects, a hover spread may be performed to control otheractions. Note that device 300 is not displaying any objects. Thisillustrates that the spread may be used to exercise other, non-objectcentric control. For example, the multiple hover point spread gesturemay be used to control broadcast power, social circle size for anotification or post, volume, intensity, or other non-object.

FIG. 21 illustrates an example z distance 2120 and z directionassociated with an example apparatus 2100 configured to perform multiplehover point gestures. The z distance may be perpendicular to apparatus2100 and may be determined by how far the tip of finger 2110 is locatedfrom apparatus 2100. While a single finger 2110 is illustrated, a zdistance may be computed for multiple hover points in a hover zone.Additionally, whether the z distance is increasing (e.g., finger movingaway from apparatus 2100) or decreasing (e.g., finger moving towardapparatus 2100) may be computed. Additionally, the rate at which the zdistance is changing may be computed. Thus, unlike conventional twofinger touch gestures that may change a parameter in a single dimension,multiple hover point gestures may operate in one, two, three, or evenfour dimensions. Consider a multiple hover point crank gesture performedusing two fingers and a thumb above a virtual screwdriver displayed on adevice. The crank gesture may not only cause the virtual screwdriver torotate in the x and y plane, but the rate at which the fingers arerotating may control how quickly the screwdriver is turned and the rateat which the fingers are approaching the screen may control the virtualpressure to be applied to the virtual screwdriver. Being able to controldirection, rate, and pressure may provide a richer user interfaceexperience than a simple one dimensional adjustment.

FIG. 22 illustrates an example displacement in an x-y direction from aninitial point 2220. Finger 2210 may initially have been located aboveinitial point 2220. Finger 2210 may then have moved to be abovesubsequent point 2230. In one embodiment, the locations of points 2220and 2230 may be described by (x,y,z) co-ordinates. In anotherembodiment, the subsequent point 2230 may be described in relation toinitial point 2220. For example, a distance, an angle in the x-ydirection, and an angle in the z direction may be employed. While asingle finger 2210 is illustrated, example apparatus and methods maytrack the displacement of multiple hover points. The tracks of themultiple hover points may facilitate identifying a gesture.

Hover technology is used to detect an object in a hover-space. “Hovertechnology” and “hover-sensitive” refer to sensing an object spaced awayfrom (e.g., not touching) yet in close proximity to a display in anelectronic device. “Close proximity” may mean, for example, beyond 1 mmbut within 1 cm, beyond 0.1 mm but within 10 cm, or other combinationsof ranges. Being in close proximity includes being within a range wherea proximity detector can detect and characterize an object in thehover-space. The device may be, for example, a phone, a tablet computer,a computer, or other device. Hover technology may depend on a proximitydetector(s) associated with the device that is hover-sensitive. Exampleapparatus may include the proximity detector(s).

FIG. 5 illustrates a hover-sensitive i/o interface 500. Line 520represents the outer limit of the hover-space associated withhover-sensitive i/o interface 500. Line 520 is positioned at a distance530 from i/o interface 500. Distance 530 and thus line 520 may havedifferent dimensions and positions for different apparatus depending,for example, on the proximity detection technology used by a device thatsupports i/o interface 500.

Example apparatus and methods may identify objects located in thehover-space bounded by i/o interface 500 and line 520. Example apparatusand methods may also identify gestures performed in the hover-space. Forexample, at a first time T1, an object 510 may be detectable in thehover-space and an object 512 may not be detectable in the hover-space.At a second time T2, object 512 may have entered the hover-space and mayactually come closer to the i/o interface 500 than object 510. At athird time T3, object 510 may retreat from i/o interface 500. When anobject enters or exits the hover-space an event may be generated.Example apparatus and methods may interact with events at this granularlevel (e.g., hover enter, hover exit, hover move) or may interact withevents at a higher granularity (e.g., hover gather, hover spread).Generating an event may include, for example, making a function call,producing an interrupt, updating a value in a computer memory, updatinga value in a register, sending a message to a service, sending a signal,or other action that identifies that an action has occurred. Generatingan event may also include providing descriptive data about the event.For example, a location where the event occurred, a title of the event,and an object involved in the object may be identified.

In computing, an event is an action or occurrence detected by a programthat may be handled by the program. Typically, events are handledsynchronously with the program flow. When handled synchronously, theprogram may have a dedicated place where events are handled. Events maybe handled in, for example, an event loop. Typical sources of eventsinclude users pressing keys, touching an interface, performing agesture, or taking another user interface action. Another source ofevents is a hardware device such as a timer. A program may trigger itsown custom set of events. A computer program that changes its behaviorin response to events is said to be event-driven.

FIG. 6 illustrates actions, objects, and data associated with a multiplehover point gesture. Region 470 provides a side view of an object 410and an object 412 that are within the boundaries of a hover-spacedefined by a distance 420 above a hover-sensitive i/o interface 400.Region 480 illustrates a top view of representations of regions of thei/o sensitive interface 400 that are affected by object 410 and object412. The solid shading of certain portions of region 480 indicates thata hover point is associated with the solid area. Region 490 illustratesa top view representation of a display that may appear on a graphicaluser interface associated with hover-sensitive i/o interface 400. Dashedcircle 430 represents a hover point graphic that may be displayed inresponse to the presence of object 410 in the hover-space and dashedcircle 432 represents a hover point graphic that may be displayed inresponse to the presence of object 412 in the hover-space. While twohover points have been detected, a user interface state or gesture statemay not transition to a multiple hover point gesture start state untilsome identifiable motion is performed by one or more of the identifiedhover points. In one embodiment, the dashed circles may be displayed oninterface 400 while in another embodiment the dashed circles may not bedisplayed. Unlike conventional systems, the hover gesture may be a purehover detect gesture that begins without touching the interface 400,without using a camera, and without reference to any particular itemdisplayed on interface 400.

FIG. 7 illustrates actions, objects, and data associated with a multiplehover point gesture. Object 410 and object 412 have moved closertogether. Region 480 now illustrates the two solid regions thatcorrespond to the two hover points associated with object 410 and 412 asbeing closer together. Region 490 now illustrates circle 430 and circle432 as being closer together. In one embodiment, circle 430 and circle432 may be displayed while in another embodiment circle 430 and circle432 may not be displayed. Example apparatus and methods may haveidentified that multiple hover points were produced in FIG. 6. The hoverpoints may have been characterized when identified. Over time, exampleapparatus and methods may have tracked the hover points and repeated thecharacterizations. The tracking and characterization may have been eventdriven. Based on the relative motion of the hover points, a multi-pointgather gesture may be identified.

Region 490 also illustrates an object 440. Object 440 may be a graphic,icon, or other representation of an item displayed by i/o interface 400.Since object 440 has been bracketed by the hover points produced byobject 410 and object 412, object 440 may be a target for a multi hoverpoint gesture. The appearance of object 440 may be manipulated toindicate that object 440 is the target of a gesture. If the distancebetween the hover point associated with circle 430 and the object 440and the distance between the hover point associated with circle 432 andthe object 440 are within gesture thresholds, then the user interface orgesture state may be changed to indicate that a certain gesture (e.g.,hover gather) is in progress. While a conventional pinch may operateonly on a single object 440 and may require an object to be disposedbetween touch points, example apparatus and methods are not so limitedand may produce a control gather event regardless of whether an objectis disposed between the hover points 430 and 432. This type ofnon-object gather may be used to control an attribute of an apparatus(e.g., reduce transmit power, enter airplane mode) rather than shrinkingan object displayed on interface 400.

FIG. 8 illustrates actions, objects, and data associated with a multiplehover point gesture. While FIGS. 6 and 7 illustrated two objects, FIG. 8illustrates three objects. Region 470 provides a side view of an object410 (e.g., finger) an object 412 (e.g., finger) and an object 414 (e.g.,thumb) that are within the boundaries of the hover-space. Region 480illustrates a top view of representations of regions of the i/osensitive interface 400 that are affected by objects 410, 412, and 414.Since thumb 414 is larger than fingers 410 and 412, the representationof thumb 414 is larger. Region 490 illustrates a top view representationof a display that may appear on a graphical user interface associatedwith hover-sensitive i/o interface 400. Dashed circle 430 represents ahover point graphic that may be displayed in response to the presence ofobject 410 in the hover-space, dashed circle 432 represents a hoverpoint graphic that may be displayed in response to the presence ofobject 412 in the hover-space, and larger dashed circle 434 represents ahover point graphic that may be displayed in response to the presence ofobject 414 in the hover-space. The objects 410, 412, and 414 may becharacterized based, at least in part, on their actual size or relativesizes. Some multiple hover point gestures may depend on using a fingerand a thumb and thus identifying which object is likely the thumb andwhich is likely the finger may be part of identifying a multiple hoverpoint gesture.

FIG. 9 illustrates actions, objects, and data associated with a multiplehover point gesture. Objects 410, 412, and 414 have moved closertogether. The hover points associated with objects 410, 412, and 414have also moved closer together. Region 490 illustrates that circles430, 432, and 434 have also moved closer together. If objects 410, 412,and 414 have moved close enough together within a short enough period oftime, then the user interface or gesture state may transition to a multihover point gather gesture detected state. If a user waits too long tomove objects 410, 412, and 414 together, or if the objects are notpositioned appropriately, then the transition may not occur. Instead,the user interface state or gesture state may transition to a gestureend state. Unlike a pinch where two points are moved together, amultiple hover point gather gesture may be defined by bringing three ormore points together. Using two points only allows defining a line.Using three points allows defining an area or a volume. Thus, the threehover points 430, 432, and 434 may define an ellipse, an ellipsoid, orother area or volume. The gather gesture may move objects located in theellipse together towards a focal point of the ellipse. Which focal pointis selected as the gather point may depend, for example, on the relativemotion of the points describing the ellipse. When four hover points areused, a rectangular or other space may be described and objects in therectangular space may be collapsed towards the center of the rectangle.Unlike conventional systems that only operate on objects and thatrequire objects to be in a pinch zone, example apparatus and methods arenot so limited. Instead, an example gather gesture may produce a gathercontrol event regardless of whether there are objects displayed anywhereon interface 400, let alone in an area or volume defined by the hoverpoints. Thus, an example multiple hover point gather gesture may be usedto control a device, a portion of a device (e.g., speaker, transmitter,radio), an interface, or other device or process independent of what isrepresented on interface 400. Additionally, unlike conventional systemsthat can only “release” an object that was pinched using a gesture thatat one point required a touch action, an example multiple hover pointspread gesture does not require a predecessor touch. For example, afarming game may be configured so that the spread gesture automaticallyspreads seed or fertilizer without having to first touch a virtualrepresentation of a seed bag or fertilizer bag.

FIG. 10 illustrates actions, objects, and data associated with amultiple hover point gesture. Objects 410, 412, and 414 have movedcloser together. Objects 410, 412, and 414 have also moved farther awayfrom the interface 400. The hover points associated with objects 410,412, and 414 have also moved closer together. Region 490 illustratesthat circles 430, 432, and 434 have also moved closer together but haveshrunk to represent the movement away from the interface 400.

Not only are the hover points associated with the objects 410, 412, 414,and 416 converging towards a focal point of an ellipse described by thethree points, but the points are also retreating from the interface 400.Unlike a conventional system that could only collapse two pointstogether along a line, the three point gather gesture may collect itemsin an area. Unlike the conventional system that could only operate onone plane, the three point gather gesture may “lift” objects in the zdirection at the same time the objects in the ellipse are gatheredtogether. Consider an application that displays photos. A user may wishto collect a set of photos together and place them in a folder.Conventionally, a user may have to select all the photos and thenperform a separate action to move the photos. Using the multiple hoverpoint gather gesture with a retreating action, the user may collect thephotos and place them in another location in a single gesture. This mayreduce memory requirements for a user interface, reduce processingrequirements for moving a collection of items, and reduce the timerequired to perform this action.

FIG. 11 illustrates actions, objects, and data associated with amultiple hover point crank gesture. Fingers 410 and 412 are located inthe hover-space associated with i/o interface 400. Thumb 414 is alsolocated in the hover-space. The hover points 430, 432, and 434associated with objects 410, 412, and 414 are illustrated in region 490.The objects 410, 412, and 414 may be characterized when they aredetected. When the objects 410, 412, and 414 move the movements of hoverpoints 430, 432, and 434 may be tracked. The tracking may be performedin response to hover point move events. If the objects 410, 412, and 414move in an identifiable pattern, then a gesture may recognized. Forexample, if the hover points 430, 432, and 434 rotate about a centerpoint or region, then a crank gesture may be identified. The crankgesture may be performed independent of any object to be turned. Whenthe crank gesture is performed parallel to the interface 400, then thegesture may be referred to as a crank gesture. When the crank gesture isperformed perpendicular to the interface 400, then the gesture may bereferred to as a roll gesture. In one embodiment, when the axis ofrotation of the gesture is at an angle of less than forty five degreesfrom the plane of the interface 400, then the gesture may be referred toas a crank gesture. In one embodiment, when the axis of rotation of thegesture is at an angle of more than forty five degrees from the plane ofthe interface 400, then the gesture may be referred to as a rollgesture.

FIG. 12 illustrates movements of objects 410, 412, and 414 that mayproduce movement in hover points 430, 432, and 434 that may beinterpreted as a multiple hover point crank gesture. For example, themovement of object 410 to location 410A, coupled with the similar andtemporally-related movement of object 412 to location 412A and object414 to location 414A may produce a regular, identifiable clockwiserotation of the three points about an axis or central point. When thetracks of the multiple points are related in this way, a multiple hoverpoint crank gesture may be identified. Identifying the gesture mayinclude, for example, identifying paths (e.g., lines, arcs) traveled bythe objects and then determining whether the paths are similar to withina threshold and whether the paths were traveled sufficientlyconcurrently. Control may then be generated in response to the crankgesture. The control may include, for example, increasing the volume ofa music player when the crank is clockwise and reducing the volume ofthe music player when the crank is counter-clockwise. The control mayinclude, for example, twisting the top on or off of a virtual jardisplayed on an apparatus, turning a screwdriver in response to thecrank gesture, or other rotational control. The control may be exercisedwithout reference to an object displayed on interface 400.

In one embodiment, the z distance of hover points associated with acrank gesture may also be considered. For example, a cranking gesturethat is approaching the i/o interface 400 may produce a first controlwhile a cranking gesture that is retreating from the i/o interface 400may produce a second, different control. For example, in a game where auser is spinning a dreidel, teetotum, or other spinning top, the objectbeing spun may drill down into the surface or may helicopter away fromthe surface based, at least in part, on whether the crank gesture wasapproaching or retreating from the i/o interface 400. In one embodiment,the crank gesture may be part of a ratchet gesture. For example, aftercranking to the right at a first speed that exceeds a speed threshold, auser may return their fingers to the left at a second slower speed thatdoes not exceed the speed threshold. The user may then repeat crankingto the right at the first faster speed and returning to the left at thesecond slower speed. In this gesture, not only the movement of thefingers but also the speed at which the fingers move determines thegesture. Like an actual ratchet device (e.g., socket wrench), theratchet gesture may be used to perform multiple turns on an object withonly turns in one direction being applied to the object, the turns inthe opposite direction being ignored. In one embodiment, the ratchetgesture may be achieved by varying the speed at which the fingersperform the crank gesture. In another embodiment, the ratchet gesturemay be achieved by varying the width of the fingers during the crank.For example, when the fingers are at a first narrower distance (e.g., 1cm) the crank may be applied to an object while when the fingers arereturning at a second wider distance (e.g., 5 cm) the crank may not beapplied.

FIG. 13 illustrates actions, objects, and data associated with amultiple hover point spread gesture. Objects 410, 412, 414, and 416 areall located in the hover zone associated with hover sensitive i/ointerface 400. The objects 410, 412, 414, and 416 are all located closeto the interface 400. Region 480 illustrates the hover points associatedwith the objects 410, 412, 414, and 416 and region 490 illustratesdashed circles 430, 432, 434, and 436 displayed in response to thepresence of the objects 410, 412, 414, and 416. The arrows in region 490indicate that the circles 430, 432, 434, and 436 are moving outwards inresponse to objects 410, 412, 414, and 416 moving outwards. Unlike aconventional two finger touch gesture that can only spread two points ona line, example apparatus and methods facilitate spreading a twodimensional area or a three dimensional volume. In one embodiment, ifobjects 410, 412, 414, and 416 spread out but stayed at the samedistance from i/o interface 400, then an area displayed by the apparatusmay increase. In another embodiment, if objects 410, 412, 414, and 416spread out and move away from the i/o interface 400, then a volume(e.g., sphere, apple, house, bubble) displayed by the apparatus mayincrease. Being able to identify an area or a volume may provide richerexperiences in, for example, video gaming where a spell may have an areaeffect or volume effect. Rather than having to describe an area using amouse or by clicking on three points, a user may simply spread theirfingers over the area or volume they wish to have covered by the spell.Similarly, being able to identify two different types of expansion orcontraction at the same time may be employed in musical applicationswhere, for example, both the volume and the reverb of a sound may bechanged. In one embodiment, when a retreating spread gesture is combinedwith a crank gesture, volume, reverb, and another attribute (e.g.,number of different sounds to be included in a chord) may all bemanipulated simultaneously. Note that the area effect spell, volume andreverb, and crank actions can be applied without a predecessor touch andindependently of an object displayed on an apparatus.

FIG. 14 illustrates actions, objects, and data associated with amultiple hover point spread gesture. Objects 410, 412, 414, and 416 havespread apart and have moved away from interface 400. Circles 430, 432,434, and 436 have also spread apart. When multiple hover points moveapart in a similar way within a threshold period of time, then amultiple hover point spread action may be identified. When the action isidentified, an event may be generated. Or, the action may be identifiedin response to an event being handled. Control associated with thespread gesture may then be applied. For example, performing a spreadgesture over a wireless enabled device may cause the device to switchinto a transmit mode while performing a gather gesture over the devicemay cause the device to switch out of the transmit mode. Performing aspread gesture over a map may cause a zoom in while performing a gathergesture may cause a zoom out. In an art application, performing a spreadgesture over a color may blend the color into the area covered by thespread gesture. In a photographic fun game, performing a spread gestureover a portion of a photograph may cause the portion of the photographcovered by the spread to distort itself to a larger shape. Performing aretreating spread may cause the distortion to look like it has occurredin three dimensions where the image is distorted to a larger shape andpulled toward the viewer.

While multiple hover point gestures including a gather, spread, andcrank have been described, and while both approaching and retreatingvariations of these gestures have been described, other multiple hoverpoint gestures are possible. For example, a multiple hover point slingshot gesture may be performed by pinching two fingers together and thenmoving the pinched fingers away from the initial pinch point to arelease point. The displacement in the x, y, or z directions may controlthe velocity, angle, and direction at which an object that was pulledback in the sling shot may be propelled in a virtual world over whichthe gesture was performed.

More generally, example apparatus and methods may detect multiple hoverpoints, characterize those multiple hover points, track the hoverpoints, and identify a gesture from the characterization and trackingdata. Control may then be exercised based on the gesture that isidentified and the movements of the multiple hover points. The controlmay be based on factors including, but not limited to, the direction(s)in which the hover points move, the rate(s) at which the hover pointsmove, the co-ordination between the multiple hover points, the durationof the gesture, and other factors. In one embodiment, the multiple hoverpoint gestures do not involve a touch, a camera, or any particular itembeing displayed on an interface with which the gesture is performed.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a memory. These algorithmic descriptions and representationsare used by those skilled in the art to convey the substance of theirwork to others. An algorithm is considered to be a sequence ofoperations that produce a result. The operations may include creatingand manipulating physical quantities that may take the form ofelectronic values. Creating or manipulating a physical quantity in theform of an electronic value produces a concrete, tangible, useful,real-world result.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, and other terms. It should be borne in mind,however, that these and similar terms are to be associated with theappropriate physical quantities and are merely convenient labels appliedto these quantities. Unless specifically stated otherwise, it isappreciated that throughout the description, terms including processing,computing, and determining, refer to actions and processes of a computersystem, logic, processor, or similar electronic device that manipulatesand transforms data represented as physical quantities (e.g., electronicvalues).

Example methods may be better appreciated with reference to flowdiagrams. For simplicity, the illustrated methodologies are shown anddescribed as a series of blocks. However, the methodologies may not belimited by the order of the blocks because, in some embodiments, theblocks may occur in different orders than shown and described. Moreover,fewer than all the illustrated blocks may be required to implement anexample methodology. Blocks may be combined or separated into multiplecomponents. Furthermore, additional or alternative methodologies canemploy additional, not illustrated blocks.

FIG. 15 illustrates an example method 1500 associated with multiplehover point gestures performed with respect to an apparatus having aninput/output display that is hover-sensitive. Method 1500 may include,at 1510, detecting a plurality of hover points in the hover-spaceassociated with the hover sensitive input/output interface. Individualobjects in the hover space may be assigned their own hover point. In oneembodiment, the plurality of hover points may include up to ten hoverpoints. In another embodiment, the plurality of hover points may beassociated with a combination of human anatomy (e.g., fingers) andapparatus (e.g., stylus). Recall that conventional systems relied oncameras or touch sensors. In one embodiment, detecting the plurality ofhover points is performed without using a camera or a touch sensor.Instead, hover points are detected using non-camera based proximitysensors that do not need an initiating touch.

Different objects may have different positions, sizes, and movements.Therefore, method 1500 may also include, at 1520, producing independentcharacterization data for members of the plurality of hover points. Inone embodiment, the characterization data for a member of the pluralityof hover points describes an (x, y, z) position in the hover-space.Position is one attribute of an object in the hover space. Size isanother attribute of an object. Therefore, in one embodiment, thecharacterization data may also include an x length measurement of theobject and a y length measurement of the object. Gestures involvemotion. However, a gesture may not involve constant motion. For example,in a sling shot gesture, the pinch and pull portion may be separatedfrom a release portion by a pause while a user lines up their shot.Thus, in one embodiment, the characterization data may also include anamount of time the member has been at the x position, an amount of timethe member has been at the y position, and an amount of time the memberhas been at the z position. If the time exceeds a threshold, then agesture may not be detected. Some gestures are defined as involving justfingers, a single finger and a single thumb, or other combinations ofdigits, stylus, or other object. Therefore, in one embodiment, thecharacterization data may also include data describing the likelihoodthat the member is a finger, data describing the likelihood that themember is a thumb, or data describing the likelihood that the member isa portion of a hand other than a finger or thumb.

In one embodiment, the characterization data is produced without using acamera or a touch sensor. Additionally, the characterization data may beproduced without reference to an object displayed on the apparatus.Thus, unlike conventional systems where a user touches an object on ascreen and then performs a hover gesture on the selected item, method1500 may proceed without a touch on the screen and without relying onany particular item being displayed on the screen. This facilitates, forexample, controlling volume or brightness without having to consumedisplay space with a volume control or brightness control.

A gesture involves motion. Therefore, method 1500 may also include, at1530, producing independent tracking data for members of the pluralityof hover points. The tracking data facilitates determining whether theobjects, and thus the hover points associated with the objects havemoved in identifiable correlated patterns associated with a specificmultiple hover point gesture.

In one embodiment, the tracking data for a member of the plurality ofhover points describes an (x, y, z) position in the hover-space for themember. The tracking data is not only concerned with where an object islocated, but also with where the hover point has been, how quickly thehover point is moving, and how long the hover point has been moving.Thus, in one embodiment, the tracking data may include a measurement ofhow much the hover point has moved in the x, y, or z direction, and arate at which the hover point is moving in the x, y, or z direction. Thetracking data may also include a measurement of how long the hover pointhas been moving in the x direction, the y direction, or the z direction.The rate at which a hover point is moving may be used to allow thegesture to operate in four dimensions (e.g., x, y, z, time). Forexample, a crank gesture may be used to turn an object, or, moregenerally, to exert rotational control. The amount of time for which therotational control will be exercised may be a function of the rate atwhich the hover points move during the gesture.

Conventional systems may have tracked single hover points for simplegestures. Example methods and apparatus may track multiple hover pointsfor more complicated gestures. The more complicated gestures involvecoordinated movement by two or more objects. Thus, the tracking data fora hover point may describe a degree of correlation between how the hoverpoint has been moving and how other hover points have been moving. Forexample, the tracking data may store information that a first hoverpoint has moved linearly a certain amount and in a certain directionduring a time window. The tracking data may also store information thata second hover point has moved linearly a certain amount and in acertain direction during the time window. The tracking data may alsostore information that the first and second hover point have moved asimilar distance in a similar direction in the time window. Or thetracking data may store information that the first and second hoverpoint have moved a similar distance in opposite directions in the timewindow.

Like the hover points are detected without using a camera or touchsensor, the tracking data may be produced without using a camera or atouch sensor. Unlike conventional systems that are designed to onlymanipulate objects that are displayed on a device, the tracking data maybe produced without reference to an object displayed on the apparatus.Thus, the tracking data may be used to identify multiple hover pointgestures that will control the apparatus as a whole, a subsystem of theapparatus, or a process running on the apparatus, rather than just anobject displayed on the apparatus.

Method 1500 may also include, at 1540, identifying a multiple hoverpoint gesture based, at least in part, on the characterization data andthe tracking data. A multiple hover point gesture like a crank involvesthe coordinated movement of, for example, two fingers and a thumb. Themovements may be simultaneous rotational motion around an axis. Indifferent embodiments, the multiple hover point gesture may be a gathergesture, a spread gesture, a crank gesture, a roll gesture, a ratchetgesture, a poof gesture, or a sling shot gesture. Other gestures may beidentified. The identification may involve determining that a thresholdnumber of objects have moved in identifiable related paths within athreshold period of time. For example, for the gather gesture, two,three, or more objects may have to move towards a gather point alongsubstantially linear paths that would intersect. For the spread gesture,two, three, or more objects may have to move outwards from adistribution point along substantially linear paths would not intersect.For a poof gesture, two coordinated spread gestures may need to beperformed by two separate sets of hover points. For example, a user mayneed to perform a spread gesture with both the right hand and the lefthand, at the same time, and at a sufficient rate, to generate the poofgesture.

FIG. 16 illustrates an example method 1600 that is similar to method1500 (FIG. 15). For example, method 1600 includes detecting a pluralityof hover points at 1610, producing characterization data at 1620,producing tracking data at 1630, and identifying a multiple hover pointgesture at 1640. However, method 1600 also includes an additionalaction. In one embodiment, method 1600 may include, at 1650, generatinga control event based on the multiple hover point gesture. The controlevent may be directed to the apparatus as a whole, to a subsystem (e.g.,speaker) on the apparatus, to a device that the apparatus controls(e.g., game console), to a process running on the apparatus, or to othercontrolled entities. In different embodiments, the control event maycontrol whether the apparatus is turned on or off or control whether aportion of the apparatus is turned on or off. In one embodiment, thecontrol event may control a volume associated with the apparatus or abrightness associated with the apparatus. In one embodiment, the controlevent may control whether a transmitter associated with the apparatus isturned on or off, whether a receiver associated with the apparatus isturned on or off, or whether a transceiver associated with the apparatusis turned on or off. Note that these control events are not associatedwith any item displayed on the apparatus. Note also that these controlevents do not involve touch interactions with the apparatus. Even thoughthe control event can exercise control independent of an objectdisplayed by the device, in one embodiment, the control event maycontrol the appearance of an object displayed on the apparatus.Generating a control event may include, for example, writing a value toa memory or register, producing a voltage in a line, generating aninterrupt, making a procedure call through a remote procedure callportal, or other action.

While FIGS. 15 and 16 illustrate various actions occurring in serial, itis to be appreciated that various actions illustrated in FIGS. 15 and 16could occur substantially in parallel. By way of illustration, a firstprocess could handle events, a second process could generate events, anda third process could exercise control over an apparatus, process, orportion of an apparatus in response to the events. While three processesare described, it is to be appreciated that a greater or lesser numberof processes could be employed and that lightweight processes, regularprocesses, threads, and other approaches could be employed.

In one example, a method may be implemented as computer executableinstructions. Thus, in one example, a computer-readable storage mediummay store computer executable instructions that if executed by a machine(e.g., computer) cause the machine to perform methods described orclaimed herein including methods 1500 or 1600. While executableinstructions associated with the listed methods are described as beingstored on a computer-readable storage medium, it is to be appreciatedthat executable instructions associated with other example methodsdescribed or claimed herein may also be stored on a computer-readablestorage medium. In different embodiments, the example methods describedherein may be triggered in different ways. In one embodiment, a methodmay be triggered manually by a user. In another example, a method may betriggered automatically.

FIG. 17 illustrates an apparatus 1700 that supports event drivenprocessing for gestures involving multiple hover points. In one example,the apparatus 1700 includes an interface 1740 configured to connect aprocessor 1710, a memory 1720, a set of logics 1730, a proximitydetector 1760, and a hover-sensitive i/o interface 1750. Elements of theapparatus 1700 may be configured to communicate with each other, but notall connections have been shown for clarity of illustration. Thehover-sensitive input/output interface 1750 may be configured to displayan item that can be manipulated by a multiple hover point gesture. Theset of logics 1730 may be configured to manipulate the state of the itemin response to multiple hover point gestures. In one embodiment,apparatus 1700 may handle hover gestures independent of there being anitem displayed on input/output interface 1750.

The proximity detector 1760 may detect an object 1780 in a hover-space1770 associated with the apparatus 1700. The proximity detector 1760 mayalso detect another object 1790 in the hover-space 1770. In oneembodiment, the proximity detector 1760 may detect, characterize, andtrack multiple objects in the hover-space simultaneously. Thehover-space 1770 may be, for example, a three dimensional volumedisposed in proximity to the i/o interface 1750 and in an areaaccessible to the proximity detector 1760. The hover-space 1770 hasfinite bounds. Therefore the proximity detector 1760 may not detect anobject 1799 that is positioned outside the hover-space 1770. A user mayplace a digit in the hover-space 1770, may place multiple digits in thehover-space 1770, may place their hand in the hover-space 1770, mayplace an object (e.g., stylus) in the hover-space, may make a gesture inthe hover-space 1770, may remove a digit from the hover-space 1770, ortake other actions. The entry of an object into hover-space 1770 mayproduce a hover-enter event. The exit of an object from hover-space 1770may produce a hover-exit event. The movement of an object in hover-space1770 may produce a hover-move event. Example methods and apparatus mayinteract with (e.g., handle) these hover events.

Apparatus 1700 may include a hover-sensitive input/output interface1750. The hover-sensitive input/output interface 1750 may be configuredto produce a hover event associated with an object in a hover-spaceassociated with the hover-sensitive input/output interface 1750. Thehover event may be, for example, a hover enter event that identifiesthat an object has entered the hover space and describes the position,size, trajectory or other information associated with the object.

Apparatus 1700 may include a first logic 1732 that is configured tohandle the hover event. The hover event may be detected in response to asignal provided by the hover-sensitive input/output interface 1750, inresponse to an interrupt generated by the input/output interface 1750,in response to data written to a memory, register, or other location bythe input/output interface 1750, or in other ways. Thus, handling thehover event involves automatically detecting a change in a physicalitem.

In one embodiment, the first logic 1732 handles the hover event bygenerating data for the object that caused the hover event. The data mayinclude, for example, position data, path data, and tracking data. Inone embodiment, the position data may be (x, y, z) coordinate data forthe object that caused the hover event. In one embodiment, the positiondata may be angle and distance data that relates the object to areference point associated with the device. In one embodiment, theposition data may include relationships between objects in the hoverspace.

The tracking data may describe where the object that produced the hoverpoint has been. In one embodiment, the tracking data may include alinked list or other organized collection of points at which the objectthat produced the hover event has been located. In one embodiment, thetracking data may include a function that describes the trajectory takenby the object that produced the hover event. The function may bedescribed using, for example, plane geometry, solid geometry, sphericalgeometry, or other models. In one embodiment, the tracking data mayinclude a reference to other tracks taken by other objects in the hoverspace. The path data may describe where the object that produced thehover point is likely headed. In one embodiment, the path data mayinclude a set of projected points that the hover point may visit based,at least in part, on where the hover point is, where the hover point hasbeen, and the rate at which the hover point is moving. In oneembodiment, the path data may include a function that describes thetrajectory likely to be taken by the object that produced the hoverevent. The function may be described using, for example, plane geometry,solid geometry, spherical geometry, or other models.

Apparatus 1700 may include a second logic 1734 that is configured todetect a multiple hover point gesture. A multiple hover point gestureinvolves at least two hover points, where at least one of the hoverpoints moves. Since apparatus 1700 is using an event driven approach,the second logic 1734 may detect the multiple hover point gesture based,at least in part, on hover events generated by objects in thehover-space. For example, a set of hover enter events followed by aseries of hover move events that produce data that describe relatedpaths and tracks within a threshold period of time may yield a multiplehover point gesture detection. The event driven approach differs fromconventional camera based approaches that perform image processing. Theevent driven approach also differs from conventional systems thatperform constant control detecting or tracking. Rather than busy waitingfor motion or wasting resources on an object that is not moving, theevent driven approach may conserve resources by responding to motion.

In one embodiment, the second logic 1734 detects a multiple hover pointgesture by correlating movements between the two or more objects. In oneembodiment, the movements are correlated as a function of analyzing theposition data, the path data, or the tracking data. A user may be usingtwo different fingers to perform two different functions on a device.For example, a user may be using their right index finger to scrollthrough a list and may be using their left index finger to control azoom factor. Although the two fingers may both be producing events, theevents are unrelated. A multiple hover point gesture involvescoordinated action by two or more objects (e.g., fingers). Thus, thesecond logic 1734 may identify movements that happen within a gesturetime window and then determine whether the movements are related. Forexample, the second logic 1734 may determine whether the objects aremoving on intersecting paths, whether the objects are moving ondiverging paths that would intersect if traveled in the oppositedirection, whether the objects are moving in a curved path around acommon axis or region, or other relationship. When relationships arediscovered, the second logic 1734 may detect the multiple hover pointgesture.

Apparatus 1700 may include a third logic 1736 that is configured togenerate a control event associated with the multiple hover pointgesture. The control event may describe, for example, the gesture thatwas performed. Thus, the control event may be, for example, a gatherevent, a spread event, a crank event, a roll event, a ratchet event, apoof event, or a slingshot event. Generating the control event mayinclude, for example, writing a value to a memory or register, producinga voltage in a line, generating an interrupt, making a procedure callthrough a remote procedure call portal, or other action. The controlevent may be applied to the apparatus 1700 as a whole, to a portion ofthe apparatus 1700, or to another device being managed or controlled byapparatus 1700. Thus, the control event may be configured to control theapparatus, a radio associated with the apparatus, a social media circleassociated with a user of the apparatus, a transmitter associated withthe apparatus, a receiver associated with the apparatus, or a processbeing performed by the apparatus. By way of illustration, a spreadgesture may be used to control the breadth of the social circle to whicha text message is to be sent. A fast wide spread gesture may send thetext to the public while a slow narrow spread gesture may only send thetext message to close friends.

Unlike conventional systems that rely on touches or cameras, the firstlogic 1732, the second logic 1734, and the third logic 1736 may operatewithout referencing touch sensor data and without referencing cameradata.

Apparatus 1700 may include a memory 1720. Memory 1720 can includenon-removable memory or removable memory. Non-removable memory mayinclude random access memory (RAM), read only memory (ROM), flashmemory, a hard disk, or other memory storage technologies. Removablememory may include flash memory, or other memory storage technologies,such as “smart cards.” Memory 1720 may be configured to store userinterface state information, characterization data, object data, dataabout the item, data about a multiple hover point gesture, data about ahover event, data about a gesture event, data associated with a statemachine, or other data.

Apparatus 1700 may include a processor 1710. Processor 1710 may be, forexample, a signal processor, a microprocessor, an application specificintegrated circuit (ASIC), or other control and processing logiccircuitry for performing tasks including signal coding, data processing,input/output processing, power control, or other functions. Processor1710 may be configured to interact with logics 1730 that handle multiplehover point gestures.

In one embodiment, the apparatus 1700 may be a general purpose computerthat has been transformed into a special purpose computer through theinclusion of the set of logics 1730. The set of logics 1730 may beconfigured to perform input and output. Apparatus 1700 may interact withother apparatus, processes, and services through, for example, acomputer network.

FIG. 18 illustrates another embodiment of apparatus 1700 (FIG. 17). Thisembodiment of apparatus 1700 includes a fourth logic 1738 that isconfigured to manage a state machine associated with the multiple hoverpoint gesture, where managing the state machine includes transitioning aprocess or data structure from a first multiple hover point state to asecond, different multiple hover point state in response to detecting aportion of a multiple hover point gesture. In one embodiment, the statemachine may include an object that stores data about the progress madein identifying or handling a multiple hover point gesture. In oneembodiment, the state machine may include a set of objects withdifferent objects associated with the different states. The statemachine may include an event handler that catches hover events orgesture events as they are generated and that updates the data, memory,objects, or processes associated with the gesture.

FIG. 19 illustrates an example cloud operating environment 1900. A cloudoperating environment 1900 supports delivering computing, processing,storage, data management, applications, and other functionality as anabstract service rather than as a standalone product. Services may beprovided by virtual servers that may be implemented as one or moreprocesses on one or more computing devices. In some embodiments,processes may migrate between servers without disrupting the cloudservice. In the cloud, shared resources (e.g., computing, storage) maybe provided to computers including servers, clients, and mobile devicesover a network. Different networks (e.g., Ethemet, Wi-Fi, 802.x,cellular) may be used to access cloud services. Users interacting withthe cloud may not need to know the particulars (e.g., location, name,server, database) of a device that is actually providing the service(e.g., computing, storage). Users may access cloud services via, forexample, a web browser, a thin client, a mobile application, or in otherways.

FIG. 19 illustrates an example multiple hover point gesture service 1960residing in the cloud. The multiple hover point gesture service 1960 mayrely on a server 1902 or service 1904 to perform processing and may relyon a data store 1906 or database 1908 to store data. While a singleserver 1902, a single service 1904, a single data store 1906, and asingle database 1908 are illustrated, multiple instances of servers,services, data stores, and databases may reside in the cloud and may,therefore, be used by the multiple hover point gesture service 1960.

FIG. 19 illustrates various devices accessing the multiple hover pointgesture service 1960 in the cloud. The devices include a computer 1910,a tablet 1920, a laptop computer 1930, a personal digital assistant1940, and a mobile device (e.g., cellular phone, satellite phone) 1950.It is possible that different users at different locations usingdifferent devices may access the multiple hover point gesture service1960 through different networks or interfaces. In one example, themultiple hover point gesture service 1960 may be accessed by a mobiledevice (e.g., phone 1950). In another example, portions of multiplehover point gesture service 1960 may reside on a phone 1950. Multiplehover point gesture service 1960 may perform actions including, forexample, producing events, handling events, updating a display,recording events and corresponding display updates, or other action. Inone embodiment, multiple hover point gesture service 1960 may performportions of methods described herein (e.g., method 1500, method 1600).

FIG. 20 is a system diagram depicting an exemplary mobile device 2000that includes a variety of optional hardware and software components,shown generally at 2002. Components 2002 in the mobile device 2000 cancommunicate with other components, although not all connections areshown for ease of illustration. The mobile device 2000 may be a varietyof computing devices (e.g., cell phone, smartphone, handheld computer,Personal Digital Assistant (PDA), etc.) and may allow wireless two-waycommunications with one or more mobile communications networks 2004,such as a cellular or satellite networks.

Mobile device 2000 can include a controller or processor 2010 (e.g.,signal processor, microprocessor, application specific integratedcircuit (ASIC), or other control and processing logic circuitry) forperforming tasks including signal coding, data processing, input/outputprocessing, power control, or other functions. An operating system 2012can control the allocation and usage of the components 2002 and supportapplication programs 2014. The application programs 2014 can includemobile computing applications (e.g., email applications, calendars,contact managers, web browsers, messaging applications), gesturehandling applications, or other computing applications.

Mobile device 2000 can include memory 2020. Memory 2020 can includenon-removable memory 2022 or removable memory 2024. The non-removablememory 2022 can include random access memory (RAM), read only memory(ROM), flash memory, a hard disk, or other memory storage technologies.The removable memory 2024 can include flash memory or a SubscriberIdentity Module (SIM) card, which is known in GSM communication systems,or other memory storage technologies, such as “smart cards.” The memory2020 can be used for storing data or code for running the operatingsystem 2012 and the applications 2014. Example data can include hoverpoint data, user interface element state, web pages, text, images, soundfiles, video data, or other data sets to be sent to or received from oneor more network servers or other devices via one or more wired orwireless networks. The memory 2020 can store a subscriber identifier,such as an International Mobile Subscriber Identity (IMSI), and anequipment identifier, such as an International Mobile EquipmentIdentifier (IMEI). The identifiers can be transmitted to a networkserver to identify users or equipment.

The mobile device 2000 can support one or more input devices 2030including, but not limited to, a touchscreen 2032, a hover screen 2033,a microphone 2034, a camera 2036, a physical keyboard 2038, or trackball2040. The mobile device 2000 may also support output devices 2050including, but not limited to, a speaker 2052 and a display 2054. Otherpossible input devices (not shown) include accelerometers (e.g., onedimensional, two dimensional, three dimensional). Other possible outputdevices (not shown) can include piezoelectric or other haptic outputdevices. Some devices can serve more than one input/output function. Forexample, touchscreen 2032 and display 2054 can be combined in a singleinput/output device.

The input devices 2030 can include a Natural User Interface (NUI). AnNUI is an interface technology that enables a user to interact with adevice in a “natural” manner, free from artificial constraints imposedby input devices such as mice, keyboards, remote controls, and others.Examples of NUI methods include those relying on speech recognition,touch and stylus recognition, gesture recognition (both on screen andadjacent to the screen), air gestures, head and eye tracking, voice andspeech, vision, touch, gestures, and machine intelligence. Otherexamples of a NUI include motion gesture detection usingaccelerometers/gyroscopes, facial recognition, three dimensional (3D)displays, head, eye, and gaze tracking, immersive augmented reality andvirtual reality systems, all of which provide a more natural interface,as well as technologies for sensing brain activity using electric fieldsensing electrodes (electro-encephalogram (EEG) and related methods).Thus, in one specific example, the operating system 2012 or applications2014 can comprise speech-recognition software as part of a voice userinterface that allows a user to operate the device 2000 via voicecommands. Further, the device 2000 can include input devices andsoftware that allow for user interaction via a user's spatial gestures,such as detecting and interpreting gestures to provide input to anapplication. In one embodiment, the multiple hover point gesture may berecognized and handled by, for example, changing the appearance orlocation of an item displayed on the device 2000.

A wireless modem 2060 can be coupled to an antenna 2091. In someexamples, radio frequency (RF) fitters are used and the processor 2010need not select an antenna configuration for a selected frequency band.The wireless modem 2060 can support two-way communications between theprocessor 2010 and external devices. The modem 2060 is shown genericallyand can include a cellular modem for communicating with the mobilecommunication network 2004 and/or other radio-based modems (e.g.,Bluetooth 2064 or Wi-Fi 2062). The wireless modem 2060 may be configuredfor communication with one or more cellular networks, such as a Globalsystem for mobile communications (GSM) network for data and voicecommunications within a single cellular network, between cellularnetworks, or between the mobile device and a public switched telephonenetwork (PSTN). Mobile device 2000 may also communicate locally using,for example, near field communication (NFC) element 2092.

The mobile device 2000 may include at least one input/output port 2080,a power supply 2082, a satellite navigation system receiver 2084, suchas a Global Positioning System (GPS) receiver, an accelerometer 2086, ora physical connector 2090, which can be a Universal Serial Bus (USB)port, IEEE 1394 (FireWire) port, RS-232 port, or other port. Theillustrated components 2002 are not required or all-inclusive, as othercomponents can be deleted or added.

Mobile device 2000 may include a multiple hover point gesture logic 2099that is configured to provide a functionality for the mobile device2000. For example, multiple hover point gesture logic 2099 may provide aclient for interacting with a service (e.g., service 1960, FIG. 19).Portions of the example methods described herein may be performed bymultiple hover point gesture logic 2099. Similarly, multiple hover pointgesture logic 2099 may implement portions of apparatus described herein.

The following includes definitions of selected terms employed herein.The definitions include various examples or forms of components thatfall within the scope of a term and that may be used for implementation.The examples are not intended to be limiting. Both singular and pluralforms of terms may be within the definitions.

References to “one embodiment”, “an embodiment”, “one example”, and “anexample” indicate that the embodiment(s) or example(s) so described mayinclude a particular feature, structure, characteristic, property,element, or limitation, but that not every embodiment or examplenecessarily includes that particular feature, structure, characteristic,property, element or limitation. Furthermore, repeated use of the phrase“in one embodiment” does not necessarily refer to the same embodiment,though it may.

“Computer-readable storage medium”, as used herein, refers to a mediumthat stores instructions or data. “Computer-readable storage medium”does not refer to propagated signals. A computer-readable storage mediummay take forms, including, but not limited to, non-volatile media, andvolatile media. Non-volatile media may include, for example, opticaldisks, magnetic disks, tapes, and other media. Volatile media mayinclude, for example, semiconductor memories, dynamic memory, and othermedia. Common forms of a computer-readable storage medium may include,but are not limited to, a floppy disk, a flexible disk, a hard disk, amagnetic tape, other magnetic medium, an application specific integratedcircuit (ASIC), a compact disk (CD), a random access memory (RAM), aread only memory (ROM), a memory chip or card, a memory stick, and othermedia from which a computer, a processor or other electronic device canread.

“Data store”, as used herein, refers to a physical or logical entitythat can store data. A data store may be, for example, a database, atable, a file, a list, a queue, a heap, a memory, a register, and otherphysical repository. In different examples, a data store may reside inone logical or physical entity or may be distributed between two or morelogical or physical entities.

“Logic”, as used herein, includes but is not limited to hardware,firmware, software in execution on a machine, or combinations of each toperform a function(s) or an action(s), or to cause a function or actionfrom another logic, method, or system. Logic may include a softwarecontrolled microprocessor, a discrete logic (e.g., ASIC), an analogcircuit, a digital circuit, a programmed logic device, a memory devicecontaining instructions, and other physical devices. Logic may includeone or more gates, combinations of gates, or other circuit components.Where multiple logical logics are described, it may be possible toincorporate the multiple logical logics into one physical logic.Similarly, where a single logical logic is described, it may be possibleto distribute that single logical logic between multiple physicallogics.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim.

To the extent that the term “or” is employed in the detailed descriptionor claims (e.g., A or B) it is intended to mean “A or B or both”. Whenthe Applicant intends to indicate “only A or B but not both” then theterm “only A or B but not both” will be employed. Thus, use of the term“or” herein is the inclusive, and not the exclusive use. See, Bryan A.Gamer, A Dictionary of Modem Legal Usage 624 (2d. Ed. 1995).

Although the subject matter has been described in language specific tostructural features or methodological acts, it is to be understood thatthe subject matter defined in the appended claims is not necessarilylimited to the specific features or acts described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims.

What is claimed is:
 1. A method, comprising: detecting a plurality ofhover points in a hover-space associated with a hover sensitiveinput/output interface associated with an apparatus; producingindependent characterization data for members of the plurality of hoverpoints; producing independent tracking data for members of the pluralityof hover points; and identifying a multiple hover point gesture based,at least in part, on the characterization data and the tracking data. 2.The method of claim 1, where the plurality of hover points includes upto ten hover points.
 3. The method of claim 1, where detecting theplurality of hover points is performed without using a camera or a touchsensor.
 4. The method of claim 1, where the characterization data for amember of the plurality of hover points describes an x position in thehover-space for the member, a y position in the hover-space for themember, a z position in the hover-space for the member, an x lengthmeasurement for the member, a y length measurement for the member, anamount of time the member has been at the x position, an amount of timethe member has been at the y position, an amount of time the member hasbeen at the z position, a likelihood that the member is a finger, alikelihood that the member is a thumb, or a likelihood that the memberis a portion of a hand other than a finger or thumb.
 5. The method ofclaim 1, where the characterization data is produced without using acamera or a touch sensor.
 6. The method of claim 1, where thecharacterization data is produced without reference to an objectdisplayed on the apparatus.
 7. The method of claim 1, where the trackingdata for a member of the plurality of hover points describes an xposition in the hover-space for the member, a y position in thehover-space for the member, a z position in the hover-space for themember, an x movement amount for the member, a y movement amount for themember, a z movement amount for the member, an x motion rate for themember, a y motion rate for the member, a z motion rate for the member,an x motion duration for the member, a y motion duration for the member,or a z motion duration for the member.
 8. The method of claim 1, wherethe tracking data for a member of the plurality of hover pointsdescribes a correlation between movement of the member and movement ofone or more other members of the plurality.
 9. The method of claim 1,where the tracking data is produced without using a camera or a touchsensor and where the tracking data is produced without reference to anobject displayed on the apparatus.
 10. The method of claim 1, where themultiple hover point gesture is a gather gesture, a spread gesture, acrank gesture, a roll gesture, a ratchet gesture, a poof gesture, or asling shot gesture.
 11. The method of claim 1, comprising generating acontrol event based on the multiple hover point gesture.
 12. The methodof claim 11, where the control event controls whether the apparatus isturned on or off, controls whether a portion of the apparatus is turnedon or off, controls a volume associated with the apparatus, controls abrightness associated with the apparatus, controls whether a transmitterassociated with the apparatus is turned on or off, controls whether areceiver associated with the apparatus is turned on or off, controlswhether a transceiver associated with the apparatus is turned on or off,or controls whether an application running on the apparatus is on oroff.
 13. The method of claim 11, where the control event controls theappearance of an object displayed on the apparatus.
 14. The method ofclaim 11, comprising determining a time period for which the controlevent is to be active based, at least in part, on a rate of motiondescribed in the tracking data.
 15. A computer-readable storage mediumstoring computer-executable instructions that when executed by acomputer cause the computer to perform a method, the method comprising:detecting a plurality of hover points in a hover-space associated with ahover sensitive input/output interface associated with an apparatus,where the plurality of hover points includes up to ten hover points, andwhere detecting the plurality of hover points is performed without usinga camera or a touch sensor; producing independent characterization datafor members of the plurality of hover points, where the characterizationdata for a member of the plurality of hover points describes an xposition in the hover-space for the member, a y position in thehover-space for the member, a z position in the hover-space for themember, an x length measurement for the member, a y length measurementfor the member, an amount of time the member has been at the x position,an amount of time the member has been at the y position, an amount oftime the member has been at the z position, a likelihood that the memberis a finger, a likelihood that the member is a thumb, or a likelihoodthat the member is a portion of a hand other than a finger or thumb,where the characterization data is produced without using a camera or atouch sensor, and where the characterization data is produced withoutreference to an object displayed on the apparatus; producing independenttracking data for members of the plurality of hover points, where thetracking data for a member of the plurality of hover points describes anx position in the hover-space for the member, a y position in thehover-space for the member, a z position in the hover-space for themember, an x movement amount for the member, a y movement amount for themember, a z movement amount for the member, an x motion rate for themember, a y motion rate for the member, a z motion rate for the member,an x motion duration for the member, a y motion duration for the member,or a z motion duration for the member, where the tracking data for amember of the plurality of hover points describes a correlation betweenmovement of the member and movement of one or more other members of theplurality, where the tracking data is produced without using a camera ora touch sensor, and where the tracking data is produced withoutreference to an object displayed on the apparatus; identifying amultiple hover point gesture based, at least in part, on thecharacterization data and the tracking data, where the multiple hoverpoint gesture is a gather gesture, a spread gesture, a crank gesture, aroll gesture, a ratchet gesture, a poof gesture, or a sling shotgesture; and generating a control event based on the multiple hoverpoint gesture, where the control event controls whether the apparatus isturned on or off, controls whether a portion of the apparatus is turnedon or off, controls a volume associated with the apparatus, controls abrightness associated with the apparatus, controls whether a transmitterassociated with the apparatus is turned on or off, controls whether areceiver associated with the apparatus is turned on or off, controlswhether a transceiver associated with the apparatus is turned on or off,or controls whether an application running on the apparatus is on oroff.
 16. An apparatus, comprising: a processor, a hover-sensitiveinput/output interface configured to produce a hover event associatedwith an object in a hover-space associated with the hover-sensitiveinput/output interface; a memory configured to store data associatedwith the hover event; a set of logics configured to process eventsassociated with a multiple hover point gesture; and an interfaceconfigured to connect the processor, the hover-sensitive input/outputinterface, the memory, and the set of logics; the set of logicsincluding: a first logic configured to handle the hover event; a secondlogic configured to detect a multiple hover point gesture based, atleast in part, on two or more hover events generated by two or moreobjects in the hover-space; and a third logic configured to generate acontrol event associated with the multiple hover point gesture, wherethe first logic, second logic, and third logic operate withoutreferencing touch sensor data and without referencing camera data. 17.The apparatus of claim 16, where the first logic handles the hover eventby generating data for the object that caused the hover event, the dataincluding position data, path data, and tracking data.
 18. The apparatusof claim 17, where the second logic detects a multiple hover pointgesture by correlating movements between the two or more objects, wherethe movements are correlated as a function of analyzing the positiondata, the path data, or the tracking data.
 19. The apparatus of claim18, where the control event is a gather event, a spread event, a crankevent, a roll event, a ratchet event, a poof event, or a slingshotevent, and where the control event is configured to control theapparatus, a radio associated with the apparatus, a social media circleassociated with a user of the apparatus, a transmitter associated withthe apparatus, a receiver associated with the apparatus, or a processbeing performed by the apparatus, and where the control event includes atime period over which the control event is to exert control, where thetime period is based, at least in part, on a rate of motion described inthe path data.
 20. The apparatus of claim 19, comprising a fourth logicconfigured to manage a state machine associated with the multiple hoverpoint gesture, where managing the state machine includes transitioning aprocess or data structure from a first multiple hover point state to asecond, different multiple hover point state in response to detecting aportion of a multiple hover point gesture.